Using an Audio Stream to Identify Metadata Associated with a Currently Playing Television Program

ABSTRACT

Systems and methods for using an audio stream to identify metadata associated with a currently playing television program are disclosed. A video stream including audio description data is received. A set of information is determined from the audio description data. A request including the set of information is sent to a server remotely located from the client for additional processing. A set of instructions is received from the server. The set of instructions is determined based on the additional processing of the set of information. One or more applications are executed in accordance with the set of instructions in response to receiving the set of instructions.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/139,727, “Using an Audio Stream to Identify Metadata Associated witha Currently Playing Television Program,” filed Dec. 23, 2013, whichclaims priority to U.S. Provisional Patent Application No. 61/785,518,entitled “Using an Audio Stream to Identify Metadata Associated with aCurrently Playing Television Program”, filed on Mar. 14, 2013, and is acontinuation-in-part of U.S. patent application Ser. No. 13/927,015, nowU.S. Pat. No. 8,849,959, entitled “Systems and Methods for Using ClosedCaptions to Initiate Display of Related Content on Second DisplayDevice,” filed on Jun. 25, 2013, which is a continuation of U.S. patentapplication Ser. No. 13/171,326, now U.S. Pat. No. 8,484,313, entitled“Using a Closed Caption Stream for Device Metadata”, filed on Jun. 28,2011, which claims priority to U.S. Provisional Application Ser. No.61/490,005, filed May 25, 2011, entitled “Used a Closed Caption Steamfor Device Metadata,” all of which are hereby incorporated by referenceherein their entireties.

TECHNICAL FIELD

The present description relates generally to systems, methods, anddevices configured to present Internet-accessible content associatedwith playing television programs.

BACKGROUND

Video content sometimes includes auditory or visual messages that promptviewers to take some action relating to the content of the video. Forexample, a television commercial may prompt a user to visit a product'swebsite associated with an advertised product to obtain coupons orinformation about the product. In another example, during the broadcastof a television program or movie, a message may appear prompting aviewer to visit a website to view more information about the people,places or things depicted in or associated with the television programor movie. Even without being prompted to, many users are interested in aperson, place or thing related to the video content they are watching.Typically, to obtain information relevant to a video stream, a userwould need to visit a website using an Internet-enabled device. Existingmethods are inefficient because they require users to take some actionthat is outside the viewing experience. Also many viewers may miss orignore audio messages and visual messages that provide additionalinformation about video content, and it may be inconvenient for manyviewers to operate a computing device while viewing video content.

SUMMARY

The methods and systems described herein provide an effective way todisplay information relevant to video content while a user is viewingthe video content. For example, a commercial about tea may be displayedon a display device and while the user is viewing the commercial, aclient device connected to the display device parses the closed captioncontent to obtain a set of information, sends the set of information toa server, obtains a set of instructions from the server and performs oneor more functions in accordance with the received set of instructions.The one or more functions display information relating to the closedcaption content. In this example, the set of instructions includeinstructions to execute one or more applications such as a browser todisplay a web page with information about a particular type or brand oftea, a media player to show an instructional video on preparing tea, afeed reader application to display articles about tea, or a coupon bookapplication to present coupons for the tea that was the subject of thecommercial. The one or more applications are executed while thecommercial is being played and the one or more applications areconcurrently displayed with the commercial on the display device.

In accordance with some implementations, systems and methods areprovided to display information. A video stream including closedcaptions is received at a client device. A set of information isdetermined from the content of the closed captions. A request includingthe set of information is sent to a server remotely located from theclient device for processing. A set of instructions is received from theserver. The set of instructions includes instructions to displayinformation relating to the set of information. One or more applicationsare executed in accordance with the set of instructions in response toreceiving the set of instructions.

In accordance with some implementations, a client request including aset of information extracted from the closed caption content in a videostream is received at a server. One or more relevant identities areidentified from the set of information. A set of instructions associatedwith the set of information is determined. The set of instructions issent to the client.

Another aspect of the disclosure includes implementations of a methodcomprising at a computing device having one or more processors andmemory storing one or more programs to be executed by the one or moreprocessors: obtaining a video stream, the video stream including audiodescription data; determining a set of information from the audiodescription data, wherein the set of information includes one or moresymbols or words derived from the audio description data; sending arequest including the set of information to a server remotely locatedfrom the computing device for processing; receiving from the server aset of instructions, wherein the set of instructions includesinstructions to display content information relating to set ofinformation; and in response to receiving, and in accordance with, theset of instructions, executing one or more applications based on a typeof the content information, wherein the one or more applications includeat least one of: a web browser, a music application, a feed readerapplication, a coupon application, and a content viewer.

In some implementations, the method further comprises formatting fordisplay, adjacent to a display of the video stream, output from the oneor more applications.

In some implementations, the set of information includes non-speechinformation derived from the audio description data. In someimplementations, the set of information includes at least one symbolextracted from the audio description data. In some implementations, theset of information includes at least one sentence extracted from theaudio description data. In some implementations, the output from the oneor more applications is displayed concurrently with the video stream.

In some implementations, the output from the one or more applications isdisplayed concurrently on a second device synchronized with thecomputing device.

In some implementations, determining the set of information from theaudio description data includes applying a speech recognition techniqueto convert the audio content to text. In some implementations,determining the set of information from the audio description dataincludes converting the audio description data into text without playingthe audio description data so that the audio description data can beheard.

In some implementations, the set of information includes at least sometext extracted from the audio description data.

In some implementations, the method further comprises: transmitting acode to initiate playing of audio content included in the audiodescription data; recording at least a portion of the audio content; andextracting text from the recorded audio content.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood in greater detail, amore particular description may be had by reference to the features ofvarious implementations, some of which are illustrated in the appendeddrawings. The appended drawings, however, merely illustrate the morepertinent features of the present disclosure and are therefore not to beconsidered limiting, for the description may admit to other effectivefeatures.

FIG. 1 is a block diagram illustrating a distributed client-serversystem in accordance with some implementations.

FIG. 2A is a block diagram illustrating the structure of an exemplaryserver system according to some implementations.

FIG. 2B is a block diagram illustrating the structure of an exemplarybroadcast system according to some implementations.

FIG. 3 is a block diagram illustrating the structure of an exemplaryclient device according to some implementations.

FIG. 4 is a flow chart illustrating an overview of the process ofdisplaying information.

FIGS. 5A, 5B, 5C, 5D, and 5E are exemplary screenshots according to someimplementations.

FIG. 6 is a flow chart illustrating the process of displayinginformation.

FIG. 7 is a flow chart illustrating the process of determining a set ofinstructions based on a set of information extracted from closedcaptions.

FIG. 8 is a diagram of an implementation of a client-server environment.

FIG. 9A is a schematic diagram of an example screenshot of a televisiondisplay in accordance with some implementations.

FIG. 9B is a schematic diagram of example screenshots of a televisiondisplay and a second screen client device (i.e., a tablet) in accordancewith some implementations.

FIG. 10 is a signal flow diagram illustrating an overview of the processof displaying associated content and information using audio descriptiondata included in program data.

FIG. 11 is a flow chart representation of an implementation of a methodof displaying associated content and information using audio descriptiondata included in program data.

FIG. 12 is a flow chart representation of an implementation of a methodof determining a set of instructions based on a set of informationextracted from audio description data.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, the drawings may not depict all of the components of agiven system, method or device; because, known methods, components, andcircuits have not been described in exhaustive detail for the sake ofbrevity. Finally, like reference numerals are used to denote likefeatures throughout the specification and figures.

DETAILED DESCRIPTION

The various implementations described herein include systems, methodsand/or devices that are enabled to identify and present, in real time,Internet-accessible and/or Internet-based functionality, content and/orservices associated with a particular television program concurrentlyplaying on a television or display, as a part of an interactivetelevision suite of technologies. In some implementations, the processof identifying associated content includes analyzing at least one ofclosed caption data and audio description data included in the videostream data for a media (e.g., television) program. For example, oneaspect of the disclosure is a method of identifying and presentingcontent associated with a media program by acquiring at least one ofclosed caption data and audio description data included in the videostream data for a media program, extracting text from the at least oneof closed caption data and audio description data included in the videostream data, transmitting a request for information based on at leastsome of the extracted text, and displaying the associated content and/orinformation received in response to the request.

Numerous details are described herein in order to provide a thoroughunderstanding of the example implementations illustrated in theaccompanying drawings. However, the invention may be practiced withoutthese specific details. Well-known methods, procedures, components, andcircuits have not been described in exhaustive detail so as not tounnecessarily obscure more pertinent aspects of the exampleimplementations.

FIG. 1 is a block diagram illustrating the distributed system 100implemented for some of the implementations. The client-server system100 includes: one or more client devices 102, a communication network104, a server system 106, a display device 108 and a broadcast system128. The server system 106 is coupled to the one or more client devices102 and the broadcast system 128 by the communication network 104.

The broadcast system 128 broadcasts the video streams 161 to the one ormore client devices 102. The server system 106 receives a requestcontaining a set of information derived from closed caption content,determines a set of instructions based on the set of information andsends the set of instructions to the client device 102 for execution.When executed by the client device 102, the set of instructions causethe client device 102 to display information relating to the set ofinformation extracted from the closed captions of a video stream.

According to some implementations, a video or video stream is a sequenceof images or frames representing scenes with objects in motion. A videodisplays a number of images or frames per second. For example, a videocommonly displays 30 frames per second. In contrast, a still image asused herein stands alone, is static or still and does not include motionor change as a video typically does. An image herein can be included orembedded within a larger image or a video.

Closed captioning refers to the ability to display caption text toselect members of an audience. Closed captions typically display atranscription of the audio portion of a program as it occurs or withslight delay (either verbatim or in edited form), sometimes includingnon-speech elements. The term “closed” in closed captions indicates thatnot all viewers can see the caption. Only users who choose to decode oractivate closed captions are able to view them. Closed captions aredistinguished from open captions, which are visible to all users. Closedcaptions are also distinguished from subtitles. Subtitles assume aviewer can hear but cannot understand the language or accent, sosubtitles only transcribe dialogue and some on-screen text. Closedcaptions aim to describe to the deaf and hard of hearing all significantaudio content including spoken dialogue, music, sound effects, andnon-speech information such as the identity of speakers and their mannerof speaking Closed captions are sometimes created by the content creatoror broadcaster of the content.

In some implementations, the server system 106 is implemented as asingle server system, while in other implementations it is implementedas a distributed system of multiple servers. Solely for convenience ofexplanation, the server system 106 is described below as beingimplemented on a single server system. In some implementations, thebroadcast system 128 is implemented as a single server system, while inother implementations it is implemented as a distributed system ofmultiple servers. Solely for convenience of explanation, the broadcastsystem 128 is described below as being implemented on a single serversystem.

The communication network(s) 104 can be any wired or wireless local areanetwork (LAN) and/or wide area network (WAN), such as an intranet, anextranet, or the Internet. It is sufficient that the communicationnetwork 104 provides communication capability between the one or moreclient devices 102 and the server system 106. In some implementations,the communication network 104 uses the HyperText Transport Protocol(HTTP) to transport information using the Transmission ControlProtocol/Internet Protocol (TCP/IP). HTTP permits client devices 102 toaccess various resources available via the communication network 104.The various implementations, however, are not limited to the use of anyparticular protocol.

In some implementations, the server system 106 includes a front endserver 112 that facilitates communication between the server system 106and the network 104. In some implementations, the front end server 112is configured to receive a request for a set of instructions. In someimplementations, the front end server 112 is configured to send a set ofinstructions to a requesting client device 102. In some implementations,the front end server 112 is configured to send content files or andlinks to content files or content files. In this context, the terms“content file” means any electronic file of any format including, butnot limited to, a video file, an image file, a music file, a web page,an email message, an SMS message, a content feed, an advertisement, acoupon, a playlist and an XML content file. In some implementations, thefront end server 112 is also configured to send a video stream to aclient 102.

A content feed (or channel) is a resource or service that provides alist of content items that are present, recently added, or recentlyupdated at a feed source. A content item in a content feed may includethe content associated with the item itself (the actual content that thecontent item specifies), a title (sometimes called a headline), and/or adescription of the content, a network location or locator (e.g., URL) ofthe content, or any combination thereof. For example, if the contentitem identifies a text article, the content item may include the articleitself inline, along with the title (or headline), and locator.Alternatively, a content item may include the title, description andlocator, but not the article content. Thus, some content items mayinclude the content associated with those items, while others containlinks to the associated content but not the full content of the items. Acontent item may also include additional metadata that providesadditional information about the content. The full version of thecontent may be any machine-readable data, including but not limited toweb pages, images, digital audio, digital video, Portable Content fileFormat (PDF) content files, and so forth.

In some implementations, a content feed is specified using a contentsyndication format, such as RSS. RSS is an acronym that stands for “richsite summary,” “RDF site summary,” or “Really Simple Syndication.” “RSS”may refer to any of a family of formats based on the Extensible MarkupLanguage (XML) for specifying a content feed and content items includedin the feed. In some other implementations, other content syndicationformats, such as the Atom syndication format or the VCALENDAR calendarformat, may be used to specify content feeds.

In some implementations, the server system 106 includes an identitydatabase 116 that stores one or more relevant identities and associatedmetrics. An identity can represent a person, place or thing and theassociated metrics measure the importance of the respective identity. Insome implementations, the identity database 116 is a distributeddatabase.

In some implementations, the server system 106 includes anidentification module 114 that manages and retrieves information storedin the identity database 116. The identification module 114 receivesrequests and returns identities stored in the identity database 116 thatmatch the request.

In some implementations, the server system 106 includes a user database124 that stores user data. The user data for a respective user includesa user identifier, associated content files and preferences. In someimplementations, the user database 124 is a distributed database.

In some implementations, the server system 106 includes a contentdatabase 122. In some implementations, the content database 122 storesvideos, images, music, web pages, email messages, SMS messages, contentfeeds, advertisements, coupons, playlists XML content files. In someimplementations, the content database 122 stores links to videos,images, music, web pages, email messages, SMS messages, content feeds,advertisements, coupons, playlists XML content files. In someimplementations, the content database 122 is a distributed database.

In some implementations, the server system 106 includes an instructionmodule 118 that generates a set of instructions to be executed by aclient device 102. The instruction module 118 generates a set ofinstructions based on information contained in a request 220 and/orinformation the user database 124. In some implementations, theinstruction module 118 selects one or more content files and/or one ormore links to content files based on a set of information contained in arequest. The instruction module 118 sends the one or more content filesand/or the one or more links to content files to the front end server112 for transmission to a requesting client device 102.

In some implementations, a user interacts with the server system 106 viaa client device 102. The client devices 102 may be any suitable computerdevice that is capable of connecting to the communication network 104,such as a desktop computer, laptop computer, tablet device, netbook,internet kiosk, personal digital assistant, mobile phone, gaming device,or any other device that is capable of communicating with the serversystem 106. The client devices 102 typically include one or moreprocessors, non-volatile memory such as a hard disk drive or flash driveand a display. In some implementations, the client devices 102 haveinput devices such as a keyboard and a mouse (as shown in FIG. 3). Insome implementations, the client devices 102 have touch screen displaysand/or microphones for input. In some implementations, the client deviceis connected to a projector. In some implementations, the client device102 is connected to a display device 108. In some implementations, theclient device 102 includes the display device 108. Stated in anotherway, in some implementations, the client device and the display deviceare contained in a single device. In some implementations, the displaydevice 108 is a television or a screen, such an LCD or LED display.

In some implementations, the client devices 102 receive video streams126 from one or more broadcast systems 128. In some implementations, theclient device 102 receives video streams from cable television inputs,satellite receivers, from a network connection or from an over-the-airantenna.

FIG. 2A is a block diagram illustrating a server system 106, inaccordance with one implementation of the present implementation. Theserver system 106 typically includes one or more processing units(CPU's) 202, one or more network or other communications interfaces 208,memory 206, and one or more communication buses 204 for interconnectingthese components. Memory 206 includes high-speed random access memory,such as DRAM, SRAM, DDR RAM o other random access solid state memorydevices; and may include non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid state storage devices.Memory 206 may optionally include one or more storage devices remotelylocated from the CPU(s) 202. Memory 206, including the non-volatile andvolatile memory device(s) within memory 206, comprises a computerreadable storage medium. In some implementations, memory 206 or thenon-transitory computer readable storage medium of memory 206 store thefollowing programs, modules and data structures, or a subset thereofincluding an operation system 216, a network communication module 218, aidentification module 114, a identity database 116, a instruction module118, a content database 122 and a user database 124.

The operating system 216 includes procedures for handling various basicsystem services and for performing hardware dependent tasks.

The network communication module 218 facilitates communication withother devices via the one or more communication network interfaces 208(wired or wireless) and one or more communication networks, such as theInternet, other wide area networks, local area networks, metropolitanarea networks, and so on.

The identity database 116 stores one or more identities 234. Arespective identity 234-2 includes a name 240, an importance metric 242and associated actions 244. The name 240 identifies the identity. Forexample, a name 240 could be the name of a person, place or thing. Theimportance metric 242 measures the importance of the identity and isused to determine which identity among a set of identities is mostimportant. The associated actions 244 specify one or more actions suchas sending a message, displaying a message and executing an application.In some implementations, the associated action 244 specifies one or morecontent files in the content database 122.

The content database 122 stores content files and/or links to contentfiles. In some implementations, the content database 122 stores media232 such as videos, images and music. In some implementations, thecontent database 122 stores advertisements 226, coupons 226, playlists228, content feeds 230 and XML content files. In some implementations,the content database 122 stores web pages 233, email messages and SMSmessages. In some implementations, the content database 122 includeslinks to videos, images, music, web pages, email messages, SMS messages,content feeds, advertisements, coupons, playlists and XML content files.In some implementations, at the content files in the content database122 include a metric measuring the popularity of the respective contentfile.

The user database 124 stores user data 236 for one or more users. Insome implementations, the user data 236-2 for a respective user includesa user identifier 246 and associated files 248. In some implementations,user data 236-2 for a respective user includes preferences 250. The useridentifier 230 identifies a user. In some implementations, theassociated files 248 comprise a list of identifiers of content filesstored in the content database 122 that are associated with the user.The files 248 associated with the user can include any file stored inthe content database 122. In some implementations, the preferences 250include categories of information in which the user is or is notinterested. For example, a user may have no interest in sports and aninterest in science fiction. In some implementations, the preferences250 include counts for categories for interest. For example, eachcategory may include a number or score representing the number of timesa user has viewed an item associated with the category. In someimplementations, the preferences 250 indicate one or more applicationsthat the user prefers or does not prefer. For example, the preferences250 may indicate that the user prefers to have songs added to user'splaylist. In another example, the preferences 250 may indicate that theuser prefers not to receive coupon offers.

The instruction module 118 generates a set of instructions 358 based oninformation contained in a request and sends the set of instructions toa client 102.

In some implementations the instruction module 116 generates a set ofinstructions 358 in response to receiving a request 220 from a client102. In some implementations, the request 220 includes a user identifier262 and a set of information 350. The user identifier 262 identifies auser with whom the request is associated. For example, the useridentifier 262 can be an IP address associated with a client device 102or an alphanumeric value chosen by the user or assigned by the serverthat uniquely identifies the user. In some implementations, the set ofinformation 350 includes information parsed from the contents of aclosed caption stream 265.

In some implementations, the set of information 350 includes stringsconsisting of alphabet characters, numbers, symbols or any combinationthereof. In some implementations, the set of information 350 includesthe name of a person, place or thing, and additional information such asa name of an application. For example, the set of information 350 mayinclude a name of a music artist and may indicate a media application.In some implementations, the set of information 350 includes the name ofa video stream and/or the category of the video stream. For example, theset of information 350 may include the name of a movie and the categoryor genre of the movie (e.g., drama, science fiction etc.). In someimplementations, the set of information 350 includes a set ofinstructions. In some implementations, the set of instructions includeinstructions to invoke one or more applications. In someimplementations, the set of instructions include instructions to displayand/or send one or more messages. In some implementations, the messagesinclude email messages and SMS messages.

In some implementations, the instruction module 118 stores the set ofinformation 350 received in a request 220 in the user database 124, andassociates the set of information 350 with a user identified by therequest 220. In some implementations, the set of information 350contained in a request 220 from a respective user is used to determinepreferences for the respective user. For example, the set of information350 may contain category information (e.g., science fiction) of contenta respective user has viewed. In this example, the instruction module118 increments the respective category counts stored in the userpreferences 250 for the respective user.

In some implementations, the instruction module 118 generates a set ofinstructions 358 based on information contained in a request 220. Insome implementations, the instruction module 118 generates a set ofinstructions 358 based on the user preferences 250 and/or the set ofinformation 350 contained in a request 220.

In some implementations, the instruction module 116 sends at least aportion of the set of information 350 to the identification module 114to obtain one or more relevant identities 234. The identification module114 matches information contained in the set of information 350 to oneor more identities 234 in the identity database 116. In someimplementations, the identification module 114 parses the set ofinformation 350 to obtain parsed information and generates searchqueries with the parsed information. For example, a set of information350 may include a list of proper nouns and the identification module 114may construct a query with each proper noun. In some implementations,the set of information 350 includes one or more sentences, specialcharacters and/or symbols. In some implementations, the one or moresentences are used as queries to the identity database 116. In someimplementations, a predefined amount of text following a specialcharacter in the set of information 350 is used as a query to theidentity database 116. For example, if the set of information 350includes the text “#actor person x” the text “person x” would be used asa query.

The generated queries are sent to the identity database 116 to findpotential matches. In some implementations, the identification module114 filters the search results based on the importance metric 242associated with the respective identities. For example, theidentification module 114 may select identities that have an importancemetric 242 above a predefined threshold or may select the top fewidentities among a set of identities. The identification module 114sends one or more determined identities to the instruction module 118.

In some implementations, the instruction module 118 receives the one ormore determined identities from the instruction module 118 anddetermines a set of instructions 358 based on the one or more determinedidentities. The instruction module 118 selects an identity from the oneor more determined identities. For example, the instruction module 118may select the identity with the highest associated importance metric242. In some implementations, the selected identity is determined basedon preferences 250 associated with the user identified by the useridentifier 262. For example, the instruction module 118 may select oneor more sports related identities when the user's preference indicatesthat the user has an affinity for sports.

After an identity is selected, the instruction module 118 generates aset of instructions 358. In some implementations, the instruction module118 determines a set of instructions for each identity determined fromthe set of information 350. In some implementations, the set ofinstructions 358 is based on the associated action 244 for the selectedidentity. For example, if the selected identity is the name of ahistorical figure, the associated action 244 may specify showing awebpage with information about the historical figure. In someimplementations, the set of instructions 358 is based on the one or moredetermined identities and user preferences 250 for the user identifiedby the user identifier 262. For example, a user's preferences 250 mayindicate applications or actions that the user prefers or does notprefer. For example, a user may prefer that certain applications are notexecuted on the user's device. In another example, the user may preferto receive information in a content feed.

In some implementations, the instruction module 118 uses the one or moredetermined identities returned by the identification module 118 togenerate search queries to identify relevant content files in thecontent database 122. For example, if the one or more determinedidentities include the name of an actor, the instruction module 118 maydetermine content files such as web pages, videos, pictures and newsstories that the actor is associated with and determines which one wouldbe most interesting to a user based metrics associated with the contentfiles. For example, the metrics may measure popularity or relevance.

In some implementations, the set of instructions 358 includesinstructions to invoke one or more functions or applications on a clientdevice 102. In some implementations, the one or more applications areselected from the group consisting of a media application, a feed readerapplication, a browser application and a coupon book application. Insome implementations, the one or more applications are selected based onthe one or more content files identified using the relevant identities.In some implementations, the set of instructions 358 includesinstructions to invoke an application and instructions to direct theinvoked application to download one or more content files contained inthe content database 118. For example, the instruction module 118 maygenerate instructions to invoke a feeder reader device and instructionsto cause the feed reader device to download content items relating tothe actor. In another example, the instruction module 118 may generateinstructions to invoke a web browser and instructions to cause thebrowser to be navigated to a particular website (e.g., a productwebsite). Additional examples of application execution are discussed inthe discussion of FIGS. 5B, 5C, 5D, and 5E.

In some implementations, the set of instructions 358 includeinstructions to display a message on the display 313 of the clientdevice 102. In some implementations, the set of instructions 358includes instructions to send an email message or SMS message to a user.The content of the email message or SMS is related to the set ofinformation 350. For example, the message may contain a promotionaloffer relating to pronouns contained in the set of information 350.

The instruction module 118 generates and sends the set of instructions358 to a client device 102 in response to a request from the clientdevice 102. In some implementations, the instructions 358 generated bythe instruction module 118 are contained in a content feed. In someimplementations, the instructions module 118 retrieves and sends one ormore content files and/or content file links (stored in the contentdatabase 122) with the set of instructions 358.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and each of the modules orprograms corresponds to a set of instructions for performing a functiondescribed above. The set of instructions can be executed by one or moreprocessors (e.g., the CPUs 202). The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules may be combined or otherwise re-arranged invarious implementations. In some implementations, memory 206 may store asubset of the modules and data structures identified above. Furthermore,memory 206 may store additional modules and data structures notdescribed above.

Although FIG. 2A shows a server system, FIG. 2A is intended more asfunctional description of the various features which may be present in aset of servers than as a structural schematic of the implementationsdescribed herein. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated. For example, some items (e.g., operatingsystem 216 and network communication module 218) shown separately inFIG. 2A could be implemented on single servers and single items could beimplemented by one or more servers. The actual number of servers used toimplement the server system 106 and how features are allocated amongthem will vary from one implementation to another, and may depend inpart on the amount of data traffic that the system must handle duringpeak usage periods as well as during average usage periods.

FIG. 2B is a block diagram illustrating a broadcast system 128, inaccordance with one implementation of the present implementation. Thebroadcast system 128 typically includes one or more processing units(CPU's) 358, one or more network or other communications interfaces 262,memory 268, a transmission interface 266 and one or more communicationbuses 262 for interconnecting these components. Memory 268 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices; and may include non-volatilememory, such as one or more magnetic disk storage devices, optical diskstorage devices, flash memory devices, or other non-volatile solid statestorage devices. Memory 268 may optionally include one or more storagedevices remotely located from the CPU(s) 358. Memory 268, including thenon-volatile and volatile memory device(s) within memory 268, comprisesa computer readable storage medium. In some implementations, memory 268or the non-transitory computer readable storage medium of memory 268store the following programs, modules and data structures, or a subsetthereof including an operation system 270, a network communicationmodule 272, a video module 273 and a video stream database 276.

The transmission interface 266 transmits video streams via radiotransmissions, satellite transmissions or through cable lines.

The operating system 270 includes procedures for handling various basicsystem services and for performing hardware dependent tasks.

The network communication module 272 facilitates communication withother devices via the one or more communication network interfaces 264(wired or wireless) and one or more communication networks, such as theInternet, other wide area networks, local area networks, metropolitanarea networks, and so on. In some implementations, the networkcommunication module 272 transfers video streams stored in the videostream database 268 via the network interface 264.

The video stream database 274 stores video streams 276. In someimplementations, video streams 276 in the video stream database 274include video content 254 and closed captions 256.

The video module 273 sends one or more video streams 276 stored in thevideo stream database 274 to one or more client devices 102.

In some implementations, the functionality of broadcast system 108 andserver system 106 can be combined on a single server system.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and each of the modules orprograms corresponds to a set of instructions for performing a functiondescribed above. The set of instructions can be executed by one or moreprocessors (e.g., the CPUs 358). The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules may be combined or otherwise re-arranged invarious implementations. In some implementations, memory 268 may store asubset of the modules and data structures identified above. Furthermore,memory 268 may store additional modules and data structures notdescribed above.

Although FIG. 2B shows a broadcast system, FIG. 2B is intended more asfunctional description of the various features which may be present in aset of servers than as a structural schematic of the implementationsdescribed herein. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated. For example, some items (e.g., operatingsystem 270 and network communication module 272) shown separately inFIG. 2B could be implemented on single servers and single items could beimplemented by one or more servers. The actual number of servers used toimplement the broadcast system 128 and how features are allocated amongthem will vary from one implementation to another, and may depend inpart on the amount of data traffic that the system must handle duringpeak usage periods as well as during average usage periods.

FIG. 3 is a block diagram illustrating a client device 102, inaccordance with some implementations. The client device 102 typicallyincludes one or more processing units (CPU's) 302, one or more networkor other communications interfaces 308, memory 306, and one or morecommunication buses 304, for interconnecting these components. Theclient device 102 may also include a user interface comprising a displaydevice 313 and a keyboard and/or mouse (or other pointing device) 314.Memory 306 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM or other random access solid state memory devices; and mayinclude non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 306 may optionallyinclude one or more storage devices remotely located from the CPU(s)302. Memory 306, or alternatively the non-volatile memory device(s)within memory 306, comprises a computer readable storage medium. In someimplementations, the client device 102 is a portable electronic devicewith a touch screen display. In some implementations, memory 306 or thecomputer readable storage medium of memory 306 store the followingprograms, modules and data structures, or a subset thereof includingoperation system 316, network communication module 318, user interfacemodule 322, parsing module 126, applications 328 and data 342.

The client device 102 includes a video input/output 351 for inputtingand outputting video. In some implementations, the video input/output351 is configured to receive video streams from radio transmissions,satellite transmissions and cable lines. In some implementations thevideo input/output 351 is connected to a cable box. In someimplementations, the video input/output 351 is connected to a satellitedish. In some implementations, the video input/output 351 is connectedto an antenna.

In some implementations, the client device 102 includes a televisiontuner 352 for receiving and recording television video streams.

The operating system 316 includes procedures for handling various basicsystem services and for performing hardware dependent tasks.

The network communication module 318 facilitates communication withother devices via the one or more communication network interfaces 304(wired or wireless) and one or more communication networks, such as theInternet, other wide area networks, local area networks, metropolitanarea networks, and so on.

The user interface module 322 tracks user input and selections to theclient device 102.

The graphics module 320 displays the user interfaces associated with theapplications 328.

The data 342 includes video streams 344. The video streams 344 includevideo content 348, closed captions 348 a, and audio description data 348b.

In some implementations, the applications 328 include a browser 330, amedia application 332, a coupon book application 336 and a feed readerapplication 340. The browser 330 allows a user to view web pages. Themedia application 332 plays videos, music, displays images and managesplaylists 334. The feed reader application 340 displays content feeds341. The coupon book application 336 displays and storescoupons/advertisements 338.

The parsing module 126 parses closed captions in the video stream togenerate a set of information, sends the set of information to a server106, receives a set of instructions from the server 106 in response tosending the set of information and executes the set of instructions.

In some implementations, the parsing module 126 parses closed captionsin a video stream to determine a set of information 350. The parsingmodule 126 parses the content of the closed captions. For example, ifthe closed captions included the text, “hello world,” the parsing modulewould parse the text, “hello world.” In some implementations, the closedcaptions include text, numbers, symbols or any combination thereof.

In some implementations, the parsing module 126 identifies one or morespecial characters in the closed captions. The special characters may bedefined in a list that is stored in the parsing module 126. The specialcharacters may be defined by the provider or broadcaster of a videostream. In some implementations, the parsing module 126 identifies oneor more special characters and a predefined amount of text following theone or more special characters. The parsing module 126 includes thespecial characters and/or the predefined amount of text in the set ofinformation 350. In some implementations, the special characters includesymbols, numbers, text or any combination thereof. For example, abroadcaster or content provider may insert symbol “#” into a closedcaption to emphasize that text following the symbol is relevant forgenerating a set of instructions. The text following the symbol mayinclude instructions or may include information from which instructionsshould be derived. For example, a closed caption may include “#actorperson x” to indicate that person x is an actor. By inserting specialcharacters into the closed captioning of a video program, a contentprovider can provide relevant information to viewers of the videoprogram while the video program is being played.

In some implementations, the parsing module 126 identifies sentences inclosed captions and includes the identified sentences in the set ofinformation 350. For example, the parsing module 126 may identifysentences by detecting periods or other punctuation marks in the closedcaptions. The parsing module 126 includes one or more of the identifiedsentences in the set of information 350.

In some implementations, the parsing module 126 identifies one or moreproper nouns in the closed captions and includes the proper nouns in theset of information 350. For example, the parsing module 126 may selectcapitalized words in the closed captions. In some implementations, theone or more proper nouns are selected if they match one or more propernouns stored in the parsing module 126. The proper nouns may correspondto a person, place or thing. For example, the parsing module 126 mayparse the closed captions to determine the names of one or more people,the name of a product or company or the name of a country. Theinstruction module 118 can use this information to issue specificrequests to the server 106, to return a link to a web page about theperson, or place identified by the proper noun or to return a mediaplaylist of media items (e.g., songs or movies) by or about the personidentified by the proper noun.

In some implementations, the set of information 350 is sent to a server106 in a request 220 generated by the request module 352. In someimplementations, the request 356 includes a user identifier of a user ofthe client 102.

In response to sending the request 220, the parsing module 126 receivesa set of instructions 358 from a server remotely located from the client102. In some implementations, the set of instructions 358 are containedin a content feed. In some implementations the parsing module 126receives one or more content files in response to the request.

The parsing module 126 executes the set of instructions 358 in responseto receiving the set of instructions 358. In some implementations, theset of instructions 358 includes instructions to display a message onthe display (e.g., 313 and 110). The content of the message relates tothe set of information. For example, the message may offer a user aproduct or service relating to a video segment. In some implementations,the message is displayed by the message module 356. In someimplementations, the set of instructions 358 includes instructions tosend an email message or SMS message to a user associated with theclient device 103. For example, the email message or SMS message mayinclude a coupon or promotional offer. In some implementations, theemail message or SMS message is sent by the message module 356.

In some implementations, the set of instructions 350 includesinstructions to execute one or more applications 328. Examples ofapplications are discussed in the discussion of FIGS. 5B, 5C, 5D and 5E.The parsing detection module 126 executes the set of instructions 350received from the server.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and each of the modules orprograms corresponds to a set of instructions for performing a functiondescribed above. The set of instructions can be executed by one or moreprocessors (e.g., the CPUs 302). The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules may be combined or otherwise re-arranged invarious implementations. In some implementations, memory 306 may store asubset of the modules and data structures identified above. Furthermore,memory 306 may store additional modules and data structures notdescribed above.

Although FIG. 3 shows a client device, FIG. 3 is intended more asfunctional description of the various features which may be present in aclient device than as a structural schematic of the implementationsdescribed herein. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated.

Attention is now directed to FIG. 4, which is a flow diagramillustrating a process 400 of displaying information, according to someimplementations. FIG. 4 provides an overview of the methods described ingreater detail in the discussion of FIGS. 6 and 7. A broadcast system128 sends a video stream containing closed captions to a client 102(401). The video stream containing closed captions is received at aclient 102 (402). A set of information is determined from the closedcaptions (404). The set of information is sent to a server 106 remotelylocated from the client 102 for processing (406). The server 106receives the set of information (408). The server 106 identifies one ormore relevant identities from the set of information, determines a setof instructions associated with the one or more relevant identities andsends the set of instructions to the client 102 (410). The client 102receives the set of instructions from the server 106 (412). The client102 executes one or more applications in accordance with the set ofinstructions (414).

Attention is now directed to FIGS. 5A, 5B, 5C, 5D, and 5E, whichillustrate exemplary screen shots according to some implementations.FIGS. 5B, 5C, 5D and 5E illustrate applications that are executed by aclient device 102. The applications in FIGS. 5B, 5C, 5D and 5E areinvoked and controlled by a set of instructions determined based on thecontent of the closed captions 504 contained in the television program502. As discussed in greater detail in the discussion of FIG. 6, amodule (e.g., parsing module 126) on a client device 102: receives avideos steam such as a television program 502, determines a set ofinformation from the closed captions 504 of the television program 502,sends the set of information to a server (e.g., server 106) to obtain aset of instructions, receives the set of instructions from the serverand invokes one or more applications on the client device 102 inaccordance with the set of instructions. The televisions program 502 andapplications are concurrently displayed on a display of a client device102. The illustrations in FIGS. 5A, 5B, 5C, 5D and 5E should be viewedas exemplary but not restrictive in nature.

FIG. 5A illustrates a screen shot displaying a television program 502that includes closed captions 504. As shown in FIG. 5A, the content ofthe closed captions 504 is “Hello World!” which may correspond to spokendialogue in the television program 502.

FIG. 5B illustrates a feed reader application 506 displayed adjacent toa television program 502. In the context of FIG. 5B, the parsing module126 on the client device 102 receives the television program 502,determines a set of information from the closed captions 504 of thetelevision program 502, sends the set of information to a server 106 toobtain a set of instructions, receives the set of instructions from theserver 106 and invokes the feed reader application 506 on the clientdevice 102 in accordance with the set of instructions received from theserver 106. The feed reader application 506 displays a content feed 508.One or more content items 510 in the content feed 508 are selected bythe server 106 based on information derived from the closed captions 504contained in the television program 502. For example, a user may bewatching a sports related television program 502 and when thecorresponding closed captions 504 mentions a sports teams (e.g., TeamX), the feed reader application 506 is invoked and the content itemSports Headline 1 510 is displayed. In the context of FIG. 5B, thecontent item 510 is chosen by the instruction module 118 on the server106 based on information or instructions derived from the closedcaptions 504.

FIG. 5C illustrates a media player 512 displayed concurrently with atelevision program 502. In the context of FIG. 5C, the parsing module126 on the client device 102 receives the television program 502,determines a set of information from the closed captions 504 of thetelevision program 502, sends the set of information to server 106 toobtain a set of instructions, receives the set of instructions from theserver 106 and invokes the media player 512 on the client device 102 inaccordance with the set of instructions received from the server 106.The media player 512 performs one or more operations based oninformation derived from the closed captions 504 contained in thetelevision program 502. For example, the closed captions 504 of thetelevision program 502 may mention an artist and when the user views theportion of the television program 502 containing the closed captions 504that relate to the artist, the media application 512 is invoked and theuser is prompted to add a song from the artist to the user's playlist.

FIG. 5D illustrates a web browser 516 displayed adjacent to a televisionprogram 502. In the context of FIG. 5D, the parsing module 126 on theclient device 102 receives the television program 502, determines a setof information from the closed captions 504 of the television program502, sends the set of information to a server (e.g., server 106) toobtain a set of instructions, receives the set of instructions from theserver and invokes the web browser 516 on the client device 102 inaccordance with the set of instructions received from the server. Insome implementations, the browser 516 displays a web page 508 chosenbased on information derived from the closed captions 504 contained inthe television program 502. The closed captions 504 may containinformation relating to a person, place or thing. For example, theclosed captions 504 may reference the name of a historical figure andwhen the user views the portion of the closed captions 504 containingthe reference to the historical figure, the web browser 516 invokes aweb page 508 (e.g., a Wikipedia page) providing information about thehistorical figure. In another example, the closed captions 504 maymention a product during a commercial and when the user views theportion of the closed captions 504 containing the reference to theproduct, the browser 516 displays a web page 508 relating to the product(e.g., a product page). When the user views the portion of the videostream containing the closed captions 504, the browser application 516is invoked and the 506 browser is navigated to a website determinedbased on the content of the closed captions 504.

FIG. 5E illustrates a coupon book application 518. In someimplementations, the coupon book application 518 performs one or moreoperations based on information derived from the closed captions 504. Inthe context of FIG. 5E, the parsing module 126 on the client device 102receives the television program 502, determines a set of informationfrom the closed captions 504 of the television program 502, sends theset of information to a server 106 to obtain a set of instructions,receives the set of instructions from the server 106 and invokes thecoupon book application 518 on the client device 102 in accordance withthe set of instructions received from the server. For example, a usermay view a commercial about pearl milk tea and the closed captions 504may mention the product. The coupon application 518 is invoked and theuser is prompted to save a coupon to the user's coupon book while theuser is viewing the commercial,

FIG. 6 illustrates a method 600 for displaying information. Such methodis useful for broadcasters and content providers that desire to provideinformation and services relating to content while the content isplayed. Such method is useful for content viewers who desire relevantinformation about content they view.

Attention is now directed to FIG. 6 which is a flow diagram illustratinga method 600 of displaying information determined based on closedcaptions contained in a video stream. The method 600 is performed at aclient device 102 having one or more processors and memory. A videostream including closed captions is received (602). For example, thevideo stream may correspond to a television program, movie or commercialwith closed captions that relate to and are coordinated with the videocontent. In some implementations, the video stream is received by thevideo input 351, the TV tuner 352 or the network interface 308 of theclient device 102. In some implementations, the video stream is receivedfrom the broadcast system 128. For example, the video stream may bereceived over a network connection, a satellite receiver, a cable box orby an over-the-air antenna.

A set of information 350 is determined from the closed caption (604). Insome implementations, the set of information 350 includes informationfrom the content of the closed captions. For example, the set ofinformation 350 may include text from the content of the closedcaptions. In some implementations, the set of information 350 consistsof alphabet characters, numbers, symbols or any combination thereof. Theset of information 350 is determined by the parsing module 126.

In some implementations, the determining includes determining one ormore symbols and the set of information 350 includes the determined oneor more symbols (606). Stated in another way, in some implementations,the parsing module 126 parses the content of the closed captions todetermine one or more symbols and includes the determined one or moresymbols in the set of information 350. In some implementations, thedetermining includes determining one or more symbols and a predefinedamount of text following the symbol. The one or more symbols andpredefined amount of text following the symbol are included in the setof information 350. For example, a broadcaster or content provider mayinsert a “#” symbol into the closed captions of a video stream to signalthat text following the symbol is information from which instructionsshould be derived. For example, a closed caption may include “#actorperson x” to indicate that person x is an actor. In someimplementations, the text following the symbol may include instructionsor information from which instructions should be derived. For example,the parsing module 126 detects and parses the string “#browser URL” fromthe closed caption and includes the parsed string into the set ofinformation 350. The parsing module 126 sends the set of information 350including the parsed string to a server 106 and in response receives aset of instructions 358. The set of instructions 358 corresponding tothe parsed string may invoke a web browser on the client device 106 andopen a webpage corresponding to the URL in the parsed string. A contentprovider may insert special characters or symbols in the closed captionsof a video stream in order to provide information relating to the videostream. The relevant information is displayed concurrently with thevideo stream.

In some implementations, the determining includes determining one ormore sentences in the closed captions and the set of information 350includes the determined one or more sentences (608). The parsing module126 parses to content of the closed captions to determine one or moresentences and includes the determined sentences in the set ofinformation 350. For example, the parsing module 126 may detectsentences by detecting periods or other punctuation marks in the contentof the closed captions.

In some implementations, the determining includes determining one ormore proper nouns in the closed captions and the set of information 350includes the one or more pronouns (610). The parsing module 126 parsesthe content of the closed captions to determine one or more proper nounsand includes the determined one or more proper nouns in the set ofinformation 350. In some implementations, the one or more proper nounsare selected if they match one or more proper nouns stored in theparsing module 126. The proper nouns may correspond to a person, placeor thing. For example, the parsing module 126 may parse the content ofthe closed captions to determine the names of one or more people, thename of a product or company or the name of a country. In someimplementations, the parsing module 126 selects capitalized words in theclosed captions and includes the capitalized words in the set ofinformation 350.

In some implementations, the video stream is displayed and the set ofinformation 350 is determined while the video stream is being displayed(612). The video stream is displayed on the display 313 of the clientdevice 102. A request 220 including the set of information 350 is sentto a server 106 remotely located from the client 102 for processing(614). The processing determines a set of instructions 358 based on theset of information 350. The set of instructions 358 include instructionsto display information relating to the closed captions of a video streamto be displayed. The request 358 is generated and sent by the requestmodule 352.

A set of instructions 358 is received from the server (616). The set ofinstructions 358 includes instructions to display information relatingto the set of information 350 (616). In some implementations, the set ofinstructions 358 is contained in a content feed. The set of instructions258 is received by the parsing module 126.

One or more functions and/or applications are executed in accordancewith the set of instruction 358 in response to receiving the set ofinstructions 350 (616). In some implementations, the one or moreapplications are selected from the group consisting of a mediaapplication, a feed reader application, a browser application and acoupon book application (620). For example, as shown in FIG. 5C, set ofinstructions 358 may invoke a media player application 512 and promptthe user to add a song to the user's playlist. In some implementations,the video stream is displayed and the one or more applications aredisplayed adjacent to the video steam (622). For example, as shown inFIG. 5B, a feed reader application 508 is displayed concurrently with atelevision program 502. The one or more applications are discussed ingreater detail in the discussion of FIGS. 5B, 5C, 5D and 5E. The one ormore applications are executed by the parsing module 126.

In some implementations, one or more content files are received from theserver (621). The set of instructions 358 includes instructions todisplay the one or more content files and executing the one or moreapplications includes displaying the one or more received content file(621). For example, a web page may be received and the set ofinstructions 358 includes instructions to invoke a web browser todisplay the webpage.

In some implementations, the set of instructions 358 includesinstructions to display a message. The content of the message relates tothe set of information 350. For example, a URL for a product's websitemay be displayed or a snippet of information relating to a televisionprogram may be displayed. In some implementations, the set ofinstructions 358 includes instructions to send a message (e.g., email orSMS) to a user associated with the client 102. For example, the messagemay include a coupon, a link to a coupon, a song, a link to a song,information about a television program or movie and links toinformation. The message module 356 displays and sends messages.

FIG. 7 is a flow diagram illustrating a method 700 of generating a setof instructions based on information derived from closed captions. Themethod 700 is performed at a server system 106 having one or moreprocessors and memory. In some implementations, a request 220 includinga set of information 350 is received from a client 102 (702). In someimplementations, the request 220 includes a user identifier. The set ofinformation 350 was extracted from closed captions in a video stream(704). One or more relevant identities 234 are identified from the setof information 350 (704). In some implementations, a plurality ofrelevant identities are identified using the set of information 350 anda subset of the plurality of relevant identities are selected based onthe importance metric 242 associated with the relevant identities. Forexample, the identification module 114 may select the relevant identitywith the highest importance metric 242. In some implementations, the oneor more relevant identities 234 are determined in response to receivingthe request 220. The identification module 114 matches information inthe set of information 350 to one or more identities 234 in the identitydatabase 130. The one or more relevant identities 234 may correspond toa person, place or thing.

A set of instructions 358 associated with the set of information isdetermined (708). In some implementations, the set of instructions 358is determined in response to receiving the request 220. In someimplementations, the set of instructions 358 is determined based on theidentified one or more relevant identities 234 (710). In someimplementations, a relevant identity 234 includes a plurality ofassociated actions. For example, a relevant identity 234 may be the nameof a basketball player and the associated actions include displaying avideo highlight of the basketball player's most recent game ordisplaying a web page about the basketball player. The instructionmodule 118 selects one or more of the plurality of associated actionsbased on user preferences 250 for the user corresponding to the useridentifier 246. For example, the user's information may includepreferences that indicate categories of information that the user is oris not interested in. For example, a relevant identity 234 may be thename of a person who is an actor, athlete and musician and the user'spreference may indicate an affinity towards sports and music. In thisexample, the resulting set of instructions 358, in one implementation,would be related to the person's activity as a musician or athlete. Insome implementations, the user's preferences indicate applications theuser likes or dislikes. For example, a user may have an affinity towardsreceiving coupons via a coupon application and content items via a feedreader application.

In some implementations, the set of instructions 358 is determined basedon one or more content files associated with the identified one or morerelevant identities 234. In some implementations, one or more contentfiles associated with the relevant identities are determined by theidentification module 114 (714). The set of instructions 358 includeinstruction to display the one or more content files (714). In someimplementations, the identification module 114 identifies the one ormore content files by searching the content database 122 using the nameof the identity as a query. For example, one of the relevant identities234 may be the name of a product and one or more coupons may beassociated with the product. In another example, one of the relevantidentities 234 may be a musician and the identification module 114identifies a song by the musician from the content database 122. In someimplementations, the associated actions 244 for a respective identityspecifies one or more content files. The instruction module 118 searchesfor the specified content files in the content database 122 and selectsa content file in accordance with a metric associated with the contentfiles. For example, the instruction module 118 may select a contentfiled based on the popularity of the files. After the instruction module118 identifies a content file, the instruction module determines anapplication to display or play the content file and generates a set ofinstructions 358 that causes the determined application to display thecontent file.

In some implementations, the set of instructions 358 includeinstructions for executing an application selected from the groupconsisting of a media application, a feed reader application, a browserapplication and a coupon book application (712).

In some implementations, the set of instructions 358 includesinstructions to display a message on the client device 102. The contentof the message relates to the set of information 350. For example, a URLfor a product's website may be displayed or a snippet of informationrelating to a television program may be displayed. In someimplementations, the set of instructions 358 includes instructions tosend a message (e.g., email or SMS) to a user associated with theclient. The message may include a coupon, a link to a coupon, a song, alink to a song, information about a television program or movie andlinks to information. The set of instructions 358 is generated by theinstruction module 118.

The set of instructions 358 is sent to the client 102 (716). In someimplementations, the set of instructions 358 is sent in a content feed.In some implementations, one or more content files are sent along withthe one or more instructions (718). For example, a playlist, media file,advertisement or feed stored in the content database 122 may be sentalong with the set of instructions 358. The set of instructions 358 andoptionally the one or more content files are sent by the instructionmodule 118.

Each of the methods described herein may be governed by instructionsthat are stored in a non-transitory computer readable storage medium andthat are executed by one or more processors of one or more servers(e.g., server system 106). Each of the operations shown in FIGS. 6 and 7may correspond to instructions stored in a computer memory or computerreadable storage medium.

FIG. 8 is a diagram of an implementation of a client-server environment800. The client-server environment 800 illustrated in FIG. 8 is similarto and adapted from the client-server environment 100 illustrated inFIG. 1. Elements common to each Figure include common reference numbers,and only the differences between FIGS. 1 and 8 are described herein forthe sake of brevity. Moreover, while certain specific features areillustrated, those skilled in the art will appreciate from the presentdisclosure that various other features have not been illustrated for thesake of brevity and so as not to obscure more pertinent aspects of theexample implementations disclosed herein.

To that end, with reference to FIG. 8, the client-server environment 800additionally includes a set-top box 103, a second screen client device120 (e.g. a laptop, a tablet computer, smartphone, etc.), and a contentprovider 150. Additionally, the display 108 has been replaced with atelevision 108-1, and client device 102 has been replaced with clientdevice 102-2, which is configured to operate as a supplemental set-topbox. In some implementations, the client device 102-2 is atelevision-integrated device. The set-top box 103, the second screenclient device 120, the client device 102-1, the server system 106, thebroadcast system 140, and the content provider 150 are capable of beingconnected to the communication network 104 in order to exchangeinformation with one another and/or other devices and systems. Theclient-server environment 800 is merely an example provided to discussmore pertinent features of the present disclosure. Those skilled in theart will appreciate from the present disclosure that any number of suchdevices and/or systems may be provided in a client-server environment,and particular devices may be absent.

In some implementations, the second screen client device 120 (e.g. atablet computer or laptop, etc.), the set-top box 103, the TV 108-1, andthe client device 102-1 are provided in combination within, for example,a home or commercial setting. The set-top box 103 is configured toreceive and decode signals including media content from the broadcastsystem 140 or the like. In operation, the set-top box 103 decodes thesignal received from the broadcast system 140 or the like and providesaudio and video content to the TV 108-1 for display. While a TV has beenused in the illustrated example, those skilled in the art willappreciate from the present disclosure that any number of displaysdevices, including computers, laptop computers, tablet computers,smart-phones and the like, can be used to display a video stream andplay the associated audio stream. Additionally and/or alternatively, insome implementations the functions of the set-top box 103 and the TV108-1 are combined into a single device.

In some implementations, the set-top box 103 is any computer devicecapable of connecting to the communication network 104, receiving videostreams, extracting information from video streams and presenting videostreams for the display using the TV 108-1 (or another display device).In some implementations, the set-top box 103 is configured as a receiverfor cable TV and/or satellite TV, a digital video recorder (DVR), adigital media receiver, a TV tuner, a computer, and/or any other devicethat outputs TV signals. In some implementations, the TV 108-1 is aconventional TV display that is not connectable to the Internet and thatdisplays digital and/or analog TV content received via over the airbroadcasts or a satellite or cable connection.

Similarly, in some implementations the client device 102-1 is anycomputer device capable of connecting to the communication network 104and the TV 108-1. In some implementations, the client device 102-1 iscapable of receiving infrared (IR) transmissions directed to the set-topbox 103 from a remote control device (not shown). Additionally and/oralternatively, in some implementations, the client device 102-1 iscapable of wired and/or wireless communication with the client device120. In some implementations, the client device 102-1 includes one ormore processors 121, non-volatile memory 125 such as a hard disk drive,and a local database 123. In some implementations, the memory 122includes application instructions 125 a and associated data 125 b. Asdiscussed in greater detail below, the processor 121 executes the one ormore applications in accordance with a set of instructions received fromthe server system 106. The client device 102-1 may also have inputdevices such as a keyboard, a mouse and/or track-pad (not shown). Insome implementations, the client device 102-1 includes a touch screendisplay, a digital camera and/or any number of supplemental devices toadd functionality.

As is typical of televisions, the TV 108-1 includes a display 108-1 aand speakers 108-1 b. Additionally and/or alternatively, the TV 108-1can be replaced with another type of display device for presenting videocontent to a user, such as for example, a computer, a tablet device, amobile telephone, a projector, or other type of video display system.The display device can be coupled to the set-top box 103 via a wirelessor wired connection.

As used herein, a TV signal is an electrical, optical, or other type ofdata transmitting medium that includes audio and/or video componentscorresponding to a TV channel or program. In some implementations, theTV signal is a terrestrial over-the-air TV broadcast signal or a signdistributed/broadcast on a cable-system or a satellite system. In someimplementations, the TV signal is transmitted as data over a networkconnection. For example, the set-top box 103 can receive video streamsfrom an Internet connection. Audio and video components of a TV signalare sometimes referred to herein as audio signals and video signals. Insome implementations, a TV signal corresponds to a TV channel that isbeing displayed on the TV 108-1.

FIG. 9A is a schematic diagram of an example screenshot of a televisiondisplay 108-1 a of the television 108-1 that is operating to display aninformation box 910 in accordance with some implementations. While atelevision is illustrated, those skilled in the art will appreciate fromthe present disclosure that the systems and methods disclosed herein maybe used in combination with any media presentation device. In operation,with reference to FIGS. 8 and 9A, in some implementations, the clientdevice 102-1 provides display instructions and display data (i.e., adisplay signal) to the television 108-1 in addition to the displaysignal provided by the primary set-top box 103. The display instructionscause the television 108-1 to display the information box 910, providedby the display data, as visual overlay obstructing at least a portion ofthe playing television program 920 presented on the screen 108-1 a. Thedisplay signal provided by the primary set-top box 103 includes the datafor presenting the playing television program 920. In someimplementations, the display data provided by the client device 102-1includes the optical attributes of the information box 910, including,for example, the relative size, shape and color of the information box910. In some implementations, the display data also includes additionalcontent associated with a playing television program 920 obtained by theprocess described below with reference to FIGS. 10 and 11. For example,as shown in FIG. 9A, the information box 710 includes a first indicatorof search results 910-1 associated with the television program, asynopsis of the television program 911-1, and selectable hyperlinks toassociated content 911-1. In some implementations, each indicator ofsearch results is associated with at least one of search results from ageneral Internet search, a targeted search for associated news items, atargeted search for associated images, a targeted search for associatedInternet-accessible media content (e.g. videos, music, etc.), and atargeted search for associated social media content.

FIG. 9B is a schematic diagram of example screenshots of the televisiondisplay 108-1 a of the television 108 and the second screen clientdevice 120 (e.g., a tablet) in accordance with some implementations. Incontrast to the example illustrated in FIG. 9A, an information box 930,providing search results associated with the playing television program920, is displayed on the screen 120 a of the second screen client device120. In operation, with reference to both FIGS. 8 and 9B, in someimplementations, the client device 102-1 provides display instructionsand display data (i.e., a display signal) to second screen client device120. The display instructions cause the second screen client device 120to display the information box 930, provided by the display data, asvisual overlay obstructing at least a portion of the screen 120 a. Insome implementations, the display data includes the optical attributesof the information box 930, including, for example, the relative size,shape and color of the information box 930. In some implementations, theinformation box 930 is presented in combination with another application(or program) window 950, and/or another portion of the display screenincludes other display area 940 that does not include a window or thelike. In some implementations, the display data also includes additionalcontent associated with a playing television program 920 obtained by theprocess briefly described above. For example, as shown in FIG. 9B, theinformation box 930 includes a first indicator of search results 930-1associated with the television program, a synopsis of the televisionprogram 931-1, and selectable hyperlinks to associated content 931-1.

FIG. 10 is a signal flow diagram illustrating an overview of the processof displaying associated content and information using audio descriptiondata included in program data. With continued reference to FIG. 8, FIG.10 provides an overview of the methods described in greater detail belowwith reference to FIGS. 11 and 12. To that end, the broadcast system 128or content provider sends a video stream containing audio descriptiondata to at least one of the primary set-top box or the client device102-1 (1001). Those skilled in the art will appreciate from the presentdisclosure that audio description data (or video description data) isgenerally provided as an accessibility feature to assist the visuallyimpaired. Audio description data generally includes a synchronized audionarrative describing what is happening in a media program (e.g., atelevision program, sports events, play, movie, etc.) during the naturalpauses in the primary audio content included in the media program (e.g.the dialogue amongst characters), and sometimes during the primary audiocontent as required. Moreover, as noted above, in variousimplementations, the client device 102-1 is configured as one of asupplemental set-top box and/or a television-integrated device.

The video stream, including the audio description data, is received byat least one of the primary set-top box 103 and the client device 102-1(1002). In some implementations, only the set-top 103 receives the videostream, including the audio description data. In some implementationsthe client device 102-1 receives the audio description data from theprimary set-top box 103. Additionally and/or alternatively, in someimplementations, the client device 102-1 requests and receives theassociated audio description data from one of the content provider 150and the broadcast system. Additionally and/or alternatively, in someimplementations the client device 102-1 records the audio descriptiondata and primary audio content as it is played by the speakers of the108-1 b of the television 108-1. Additionally and/or alternatively, insome implementations the client device 120 (e.g., a tablet, smartphone,etc.) records the audio description data and primary audio content as itis played by the speakers of the 108-1 b of the television 108-1, andtransmits at least a portion of the recording to the client device102-1.

A set of information is determined from the audio description data(1004). In some implementations, the set of information is in partdetermined by performing an audio-to-text transcription process on theaudio description data, which converts the audio content into text byapplying one or more speech recognition techniques. In someimplementations, the set of information is in part determined byconverting the audio description data into text without playing theaudio description data so that the audio description data can be heard.For example, in addition to or as an alternative to being converted to asignal suitable for a speaker (or another sound actuator), the audiodescription data is decoded to obtain a text stream.

The set of information is sent to the server 106 remotely located fromthe client device 102-1 for processing (1006). The server 106 receivesthe set of information (1008). The server 106 identifies one or morerelevant identities from the set of information, determines a set ofinstructions associated with the one or more relevant identities andsends the set of instructions to the client device 102-1 (1010). Theclient device 102-1 receives the set of instructions from the server 106(1012). The client device 102 executes one or more applications inaccordance with the set of instructions in order to cause the display ofassociated content on the television screen 108-1 a and/or the screen ofthe client device 120 (1014).

FIG. 11 is a flow chart representation of an implementation of a method1100 of displaying associated content and information using audiodescription data included in program data. Such a method may be usefulfor broadcasters and content providers that desire to provideinformation and services relating to content while the content isplayed. Such method may be useful for content viewers who desirerelevant information about content they view. In some implementations,the method 1100 is performed by a device, such as a supplemental set-topbox and/or a television integrated module to provide content orinformation associated with the currently playing television program.For example, with reference to FIG. 8, in some implementations, themethod 1100 is performed by the client device 102-1 (e.g., asupplemental set-top box) and/or the client device 120. Briefly, themethod 1100 includes identifying and presenting content associated witha media program by acquiring audio description data associated with thevideo stream data of the media program, extracting text from the audiodescription data, transmitting a request for information based on atleast some of the extracted text, and displaying the associated contentand/or information received in response to the request.

To that end, the method 1100 includes receiving a video stream 161including audio description data (1102). For example, the video stream161 may correspond to a television program, movie or commercial withaudio description data that relate to and is synchronized with theprimary video and audio content. In some implementations, the videostream 161 is received by the video input 350, the TV tuner 352, thenetwork interface 308 of the client device 102-1, and/or the primaryset-top box 103. In some implementations, the video stream 161 isreceived from at least one of the broadcast system 128 and the contentprovider 150. In various implementations, the video stream 161 may bereceived over a network connection, a satellite receiver, a cable boxand/or by an over-the-air antenna.

The method 1100 includes determining a set of information 350 from theaudio description data (1104). In some implementations, the set ofinformation is in part determined by performing an audio-to-texttranscription process on the audio description data, which converts theaudio content into text by applying one or more speech recognitiontechniques. In some implementations, the set of information is in partdetermined by converting the audio description data into text withoutplaying the audio description data so that the audio description datacan be heard. For example, in addition to or as an alternative to beingconverted to a signal suitable for a speaker (or another soundactuator), the audio description data is decoded to obtain a textstream. In some implementations, the set of information 350 includesinformation from the content of the audio description data. For example,in some implementations, the set of information 350 includes text fromthe transcription of the audio description data. In someimplementations, the set of information 350 consists of alphabetcharacters, numbers, symbols or any combination thereof. In someimplementations, the set of information is determined by a parsingmodule (e.g., parsing 126 illustrated in FIG. 1). In someimplementations, the set of information 350 is determined by a processorexecuting a set of computer readable instructions (e.g., processor 121and computer readable instructions 125 a illustrated in FIG. 8).

In some implementations, the determining includes determining one ormore symbols and the set of information 350 includes the determined oneor more symbols (1106). In other words, in some implementations, theparsing module 126 parses the audio description data to determinewhether one or more symbols are described and includes the determinedone or more symbols in the set of information 350. In someimplementations, the determining includes determining one or moresymbols and a predefined amount of text following the symbol. The one ormore symbols and predefined amount of text following the symbol areincluded in the set of information 350. For example, a broadcaster orcontent provider may describe or include a “#” symbol in the audiodescription data of the video stream to signal that text following thesymbol is information from which instructions should be derived. Forexample, the audio description data may include “#actor person x” toindicate that person x is an actor. In some implementations, the textfollowing the symbol may include instructions or information from whichinstructions should be derived. For example, the parsing module 126detects and parses the string “#browser URL” from the audio descriptiondata and includes the parsed string in the set of information 350. Theparsing module 126 sends the set of information 350 including the parsedstring to a server 106 and in response receives a set of instructions358. The set of instructions 358 corresponding to the parsed string mayinvoke a web browser on the client device 106 and open a webpagecorresponding to the URL in the parsed string. A content provider maydescribe special characters or symbols in the audio description data ofa video stream in order to provide information relating to the videostream. The relevant information is displayed concurrently with thevideo stream.

In some implementations, the determining includes determining one ormore sentences in the audio description data and the set of information350 includes the determined one or more sentences (1108). The parsingmodule 126 parses to content of the audio description data to determineone or more sentences and includes the determined sentences in the setof information 350. For example, the parsing module 126 may detectsentences by detecting periods or other punctuation marks in the contentof the audio description data.

In some implementations, the determining includes determining one ormore proper nouns in the audio description data and the set ofinformation 350 includes the one or more pronouns (1110). The parsingmodule 126 parses the content of the audio description data to determineone or more proper nouns and includes the determined one or more propernouns in the set of information 350. In some implementations, the one ormore proper nouns are selected if they match one or more proper nounsstored in the parsing module 126. The proper nouns may correspond to aperson, place or thing. For example, the parsing module 126 may parsethe content of the audio description data to determine the names of oneor more people, the name of a product or company or the name of acountry. In some implementations, the parsing module 126 selectscapitalized words in the audio description data and includes thecapitalized words in the set of information 350.

In some implementations, the video stream is displayed and the set ofinformation 350 is determined while the video stream is being displayed(1112). The video stream is displayed on the display 313 of the clientdevice 102. A request 220 including the set of information 350 is sentto a server 106 remotely located from the client 102 for processing(1114). The processing determines a set of instructions 358 based on theset of information 350. The set of instructions 358 include instructionsto display information relating to the audio description data of a videostream to be displayed. The request 358 is generated and sent by therequest module 352.

A set of instructions 358 is received from the server (1116). The set ofinstructions 358 includes instructions to display information relatingto the set of information 350 (1116). In some implementations, the setof instructions 358 is contained in a content feed. The set ofinstructions 258 is received by the parsing module 126.

One or more functions and/or applications are executed in accordancewith the set of instruction 358 in response to receiving the set ofinstructions 350 (1118). In some implementations, the one or moreapplications are selected from the group consisting of a mediaapplication, a feed reader application, a browser application and acoupon book application (1120). For example, as shown in FIG. 5C, set ofinstructions 358 may invoke a media player application 512 and promptthe user to add a song to the user's playlist. In some implementations,one or more content files are received from the server (1121). The setof instructions 358 includes instructions to display the one or morecontent files and executing the one or more applications includesdisplaying the one or more received content file (1121). For example, aweb page may be received and the set of instructions 358 includesinstructions to invoke a web browser to display the webpage.

In some implementations, the video stream is displayed and the one ormore applications are displayed adjacent to the video steam (1122). Forexample, as shown in FIG. 5B, a feed reader application 508 is displayedconcurrently with a television program 502. The one or more applicationsare discussed in greater detail in the discussion of FIGS. 5B, 5C, 5Dand 5E. The one or more applications are executed by the parsing module126.

In some implementations, the set of instructions 358 includesinstructions to display a message. The content of the message relates tothe set of information 350. For example, a URL for a product's websitemay be displayed or a snippet of information relating to a televisionprogram may be displayed. In some implementations, the set ofinstructions 358 includes instructions to send a message (e.g., email orSMS) to a user associated with the client 102. For example, the messagemay include a coupon, a link to a coupon, a song, a link to a song,information about a television program or movie and links toinformation. The message module 356 displays and sends messages.

FIG. 12 is a flow chart representation of an implementation of a method1200 of determining a set of instructions based on a set of informationextracted from audio description data. In some implementations, themethod 1200 is performed at a server system having one or moreprocessors and memory (e.g. server system 106 of FIG. 8). Briefly, themethod 1200 includes receiving a request for content associated with amedia program, determining associated content based at least on a set ofinformation provided with the request, and transmitting associatedcontent to the requesting device.

To that end, the method 1200 includes receiving a request 220 includinga set of information 350 from the client device 102-1 or the clientdevice 120 (1202). In some implementations, the request 220 includes auser identifier. In some implementations, the set of information 350includes data extracted from audio description data associated with avideo stream (1204).

In turn, the method 1200 includes identifying one or more relevantidentities 234 from the set of information 350 (1206). In someimplementations, a plurality of relevant identities are identified usingthe set of information 350 and a subset of the plurality of relevantidentities are selected based on the importance metric 242 associatedwith the relevant identities. For example, the identification module 114may select the relevant identity with the highest importance metric 242.In some implementations, the one or more relevant identities 234 aredetermined in response to receiving the request 220. The identificationmodule 114 matches information in the set of information 350 to one ormore identities 234 in the identity database 130. The one or morerelevant identities 234 may correspond to a person, place or thing.

The method 1200 includes determining a set of instructions 358associated with the set of information (1208). In some implementations,the set of instructions 358 is determined in response to receiving therequest 220. In some implementations, the set of instructions 358 isdetermined based on the identified one or more relevant identities 234(1210). In some implementations, a relevant identity 234 includes aplurality of associated actions. For example, a relevant identity 234may be the name of a basketball player and the associated actionsinclude displaying a video highlight of the basketball player's mostrecent game or displaying a web page about the basketball player. Theinstruction module 118 selects one or more of the plurality ofassociated actions based on user preferences 250 for the usercorresponding to the user identifier 246. For example, user informationmay include preferences that indicate categories of information that aparticular user is or is not interested in. For example, a relevantidentity 234 may be the name of a person who is an actor, athlete andmusician and the user's preference may indicate an affinity towardssports and music. In this example, the resulting set of instructions358, in one implementation, would be related to the person's activity asa musician or athlete. In some implementations, the user's preferencesindicate applications the user likes or dislikes. For example, a usermay have an affinity towards receiving coupons via a coupon applicationand content items via a feed reader application.

In some implementations, the set of instructions 358 includeinstructions for executing an application selected from the groupconsisting of a media application, a feed reader application, a browserapplication and a coupon book application (1212).

In some implementations, the set of instructions 358 is determined basedon one or more content files associated with the identified one or morerelevant identities 234. In some implementations, one or more contentfiles associated with the relevant identities are determined by theidentification module 114 (1214). The set of instructions 358 includeinstruction to display the one or more content files (1214). In someimplementations, the identification module 114 identifies the one ormore content files by searching the content database 122 using the nameof the identity as a query. For example, one of the relevant identities234 may be the name of a product and one or more coupons may beassociated with the product. In another example, one of the relevantidentities 234 may be a musician and the identification module 114identifies a song by the musician from the content database 122. In someimplementations, the associated actions 244 for a respective identityspecify one or more content files. The instruction module 118 searchesfor the specified content files in the content database 122 and selectsa content file in accordance with a metric associated with the contentfiles. For example, the instruction module 118 may select a contentfiled based on the popularity of the files. After the instruction module118 identifies a content file, the instruction module determines anapplication to display or play the content file and generates a set ofinstructions 358 that causes the determined application to display thecontent file.

In some implementations, the set of instructions 358 includesinstructions to display a message on the client device 102. The contentof the message relates to the set of information 350. For example, a URLfor a product's website may be displayed or a snippet of informationrelating to a television program may be displayed. In someimplementations, the set of instructions 358 includes instructions tosend a message (e.g., email or SMS) to a user associated with theclient. The message may include a coupon, a link to a coupon, a song, alink to a song, information about a television program or movie andlinks to information. The set of instructions 358 is generated by theinstruction module 118.

The method 1200 includes sending the set of instructions 358 to theclient 102 (1216). In some implementations, the set of instructions 358is sent in a content feed. In some implementations, one or more contentfiles are sent along with the one or more instructions (1218). Forexample, a playlist, media file, advertisement or feed stored in thecontent database 122 may be sent along with the set of instructions 358.The set of instructions 358 and optionally the one or more content filesare sent by the instruction module 118.

Each of the methods described herein may be governed by instructionsthat are stored in a non-transitory computer readable storage medium andthat are executed by one or more processors of one or more servers(e.g., server system 106). Each of the operations shown in FIGS. 11 and12 may correspond to instructions stored in a computer memory orcomputer readable storage medium.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the methods and systems to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The implementations were chosen and described in order tobest explain the principles of the methods and systems and theirpractical applications, to thereby enable others skilled in the art tobest utilize the techniques and devices described herein and variousimplementations with various modifications as are suited to theparticular use contemplated.

Moreover, in the foregoing description, numerous specific details areset forth to provide a thorough understanding of the presentimplementation. However, it will be apparent to one of ordinary skill inthe art that the methods described herein may be practiced without theseparticular details. In other instances, methods, procedures, components,and networks that are well known to those of ordinary skill in the artare not described in detail to avoid obscuring aspects of the presentimplementation.

What is claimed is:
 1. A method, comprising: at a computing devicehaving one or more processors and memory storing one or more programs tobe executed by the one or more processors: obtaining a video stream, thevideo stream including audio description data; determining a set ofinformation from the audio description data, wherein the set ofinformation includes one or more symbols or words derived from the audiodescription data; sending a request including the set of information toa server remotely located from the computing device for processing;receiving from the server a set of instructions, wherein the set ofinstructions includes instructions to display content informationrelating to the set of information; and in response to receiving, and inaccordance with, the set of instructions, executing one or moreapplications based on a type of the content information, wherein the oneor more applications include at least one of: a web browser, a musicapplication, a feed reader application, a coupon application, and acontent viewer.
 2. The method of claim 1, further comprising formattingfor display, adjacent to a display of the video stream, output from theone or more applications.
 3. The method of claim 1, wherein the set ofinformation includes non-speech information derived from the audiodescription data.
 4. The method of claim 1, wherein the set ofinformation includes at least one symbol extracted from the audiodescription data.
 5. The method of claim 1, wherein the set ofinformation includes at least one sentence extracted from the audiodescription data.
 6. The method of claim 1, wherein the output from theone or more applications is displayed concurrently with the videostream.
 7. The method of claim 1, wherein the output from the one ormore applications is displayed concurrently on a second devicesynchronized with the computing device.
 8. The method of claim 1,wherein determining the set of information from the audio descriptiondata includes applying a speech recognition technique to convert theaudio content to text.
 9. The method of claim 1, wherein determining theset of information from the audio description data includes convertingthe audio description data into text without playing the audiodescription data so that the audio description data can be heard. 10.The method of claim 1, wherein the set of information includes at leastsome text extracted from the audio description data.
 11. The method ofclaim 1, further comprising: transmitting a code to initiate playing ofaudio content included in the audio description data; recording at leasta portion of the audio content; and extracting text from the recordedaudio content.
 12. A computing system, comprising one or moreprocessors; memory; and one or more programs, wherein the one or moreprograms are stored in the memory and configured to be executed by theone or more processors, the one or more programs including instructionsfor: obtaining a video stream, the video stream including audiodescription data; determining a set of information from the audiodescription data, wherein the set of information includes one or moresymbols or words derived from the audio description data; sending arequest including the set of information to a server remotely locatedfrom the computing device for processing; receiving from the server aset of instructions, wherein the set of instructions includesinstructions to display content information relating to the set ofinformation; and in response to receiving, and in accordance with, theset of instructions, executing one or more applications based on a typeof the content information, wherein the one or more applications includeat least one of: a web browser, a music application, a feed readerapplication, a coupon application, and a content viewer.
 13. The systemof claim 12, wherein the one or more programs further compriseinstructions for: formatting for display, adjacent to a display of thevideo stream, output from the one or more applications.
 14. The systemof claim 12, wherein the set of information includes non-speechinformation derived from the audio description data.
 15. The system ofclaim 12, wherein the set of information includes at least one symbolextracted from the audio description data.
 16. The system of claim 12,wherein the set of information includes at least one sentence extractedfrom the audio description data.
 17. A non-transitory computer readablestorage medium storing one or more programs configured for execution byone or more processors of a server system, the one or more programscomprising instructions, to be executed by the one or more processors,for: obtaining a video stream, the video stream including audiodescription data; determining a set of information from the audiodescription data, wherein the set of information includes one or moresymbols or words derived from the audio description data; sending arequest including the set of information to a server remotely locatedfrom the computing device for processing; receiving from the server aset of instructions, wherein the set of instructions includesinstructions to display content information relating to the set ofinformation; and in response to receiving, and in accordance with, theset of instructions, executing one or more applications based on a typeof the content information, wherein the one or more applications includeat least one of: a web browser, a music application, a feed readerapplication, a coupon application, and a content viewer.
 18. Thenon-transitory computer readable storage medium of claim 17, wherein theone or more programs further comprise instructions for: formatting fordisplay, adjacent to a display of the video stream, output from the oneor more applications.
 19. The non-transitory computer readable storagemedium of claim 17, wherein the set of information includes non-speechinformation derived from the audio description data.
 20. Thenon-transitory computer readable storage medium of claim 17, wherein theset of information includes at least one symbol extracted from the audiodescription data.